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ABSTRACT 



This paper looks at the process of assessment of a 
computer- enhanced classroom experience during the implementation phase. It 
utilizes an assessment model based on Rathbun and Goodrum (1994) that 
suggests multi-methods of data collection. The use of triangulation to answer 
a research question fits into the proposed multi -method design. This paper 
reports how assessment was used to measure the effectiveness of the 
implementation of a multimedia application for use in support of a 
traditional classroom. The Living Textbook was developed to support a senior 
level management class in a Recreation and Park Administration program. The 
instructor identified four instructional goals of the multi-media program. 
Triangulation techniques included group observations, individual student 
observations, data based and open ended surveys, and debriefing of 
instructors. The results suggest that usability/ accessibility were essential 
precursors to students developing receptivity to the multimedia program. When 
receptivity was achieved, students valued the program as a contributor to 
their base of knowledge about the real work world. The use of student 
workbooks, which allowed opportunities for learning by doing, and in-class 
discussions in small groups were strongly linked to valuing The Living 
Textbook. (Contains 10 references.) (Author/SWC) 
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Assessing the Effectiveness of a Computer- Enhanced 
Classroom 



ASSESSING THE EFFECTIVENESS OF A 
COMPUTER-ENHANCED CLASSROOM 

by Daniel D. McLean, Ph.D., Russell E. Brayley, Ph. D., and Gail Rathbun, M.A. 

Indiana University, Bloomington 



ABSTRACT 

This paper looks at the process of assessment of a computer-enhanced classroom experience 
during the implementation phase. It utilizes an assessment model based on Rathbun and 
Goodrum (1994) that suggests multi-methods of data collection. The use of triangulation 
(Denzin, 1978) to answer a research question fits into the proposed multi-method design. The 
Living Textbook was developed to support a senior level management class in a Recreation 
and Park Administration program. The instructor identified four instructional goals of the 
multi-media program. Triangulation techniques included group observations, individual 
student observations, data based and open ended surveys, and debriefing of instructors. The 
results suggested that usability/ accessibility were essential precursors to students developing 
receptivity to the multimedia program. When receptivity was achieved students valued the 
program as a contributor to their base of knowledge about the real work world. The use of 
student workbooks where opportunities for learning by doing occurred and in-class 
discussions in small groups were strongly linked to valuing the Living Textbook. 

INTRODUCTION 

The integration of technology in the classroom environment is becoming more widely 
accepted throughout postsecondary institutions of higher learning. Hutchison (1995) described 
the current period as a waning print culture — a retreat from the Gutenberg galaxy. This 
description may seem overly optimistic or pessimistic depending upon one's point of view. 
The investment in technology by higher education institutions, however, is going forward at 
a rapid pace. The assessment of the effectiveness of computer technology as a part of the 
classroom environment, however, is in its early stages. Of the 69 papers accepted for this 
conference, only 3 are included under the topic, "Evaluating On-line Instruction." The 
authors are not so naive to believe that these are the only 3 papers on evaluation, but the 
implication remains, conference presenters continue to focus on process and implementation 
at the expense of assessment. Ehrmann (1996) has suggested, "Thus far, few educators, 
evaluators, and researchers have paid much attention to educational strategies for using 
technologies. Too often they've been victims of 'rapture of technologies.’" Assessment of 
process is critically important to understanding educational outcomes and altering strategies 
to achieve desired outcomes. 

This paper reports how assessment was used to measure the effectiveness of the 
implementation of a multimedia application for use in support of a traditional classroom. 
This paper does not address the design process or assessment issues related to it. 
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THE LIVING TEXTBOOK 



The Living Textbook (LT ) (http://www.indiana.edu/~tltl/htodcs/projects/mclean.html) 
project was initially conceived as an effort to bring together the functionality of a 
computer with the needs of the classroom. The classroom it was designed for was 
already a paperless classroom. Students received all of their assignments and 
grades through the world wide web (WWW) and turned in all assignments via a 
local area network. However, the classroom lecture and discussion format 
remained the primary teaching method. One of the primary goals of the LT was to 
provide students with an interactive learning environment that would enhance 
rather than replace the existing classroom. A second goal of the LT was to build the 
individual student's base of professional experience. The third goal focused on 
enriching student understanding of management as a dynamic rather than static 
process. A fourth goal of the LT project focused on enhancing basic and 
intermediate computer skills. Using the LT as a foundation, students were also 
exposed to electronic communications software, the WWW, word processing, 
spread sheet and graphics software (McLean and Brayley, 1996). 

THE ASSESSMENT PROCESS 

Rathbun and Goodrum (1994) proposed four issues that pose obstacles to effective evaluation 
of technologically based and supported instruction. The first, continuous change, refers to the 
ongoing barrage of new technologies confronting a designer. The second issue is technology 
focus or the ability of the institution to deliver a developed multimedia program. Third, 
evaluation in isolation, refers to the frequent lack of an integrated approach to the assessment 
of design and implementation. The fourth issue is that of hyped media, or the claims made as 
to the superiority of instructional media. Each of these issues requires a level of attention in 
the development of any assessment program. Scriven (1986) has characterized traditional 
social science approaches to assessment as naive and inappropriate. The Annenberg/CPB 
Flashlight Project which Stephen Ehrmann heads has attempted to develop alternatives to 
traditional assessment strategies which address some of the obstacles and challenges 
mentioned above. 



"Ordinarily what matters most is not the technology per se but how it is used; 
not so much what happens in the moments when the student is using the 
technology, but more how those uses promote larger improvements in the 
fabric of the student's education...." (Ehrmann, 1996). 



The Annenberg/CPB Project is one effort to address technology and learning implementation 
strategy. For example, in looking at learning the project suggested that assessment needed to 
focus on whether the outcomes would produce greater engagement in learning and more 
productive time on task by students. 



An integrated model for evaluating multimedia instruction, outlined by Rathbun and 
Goodrum (1994), provided the framework for our assessment of LT. A focus on user 
satisfaction looks first at design issues such as moving about, finding things, and control of 
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tasks appropriate to the level of the user. Valuing is an essential part of this characteristic and 
states that a tool can be "viewed as relevant, critical, and of wide applicability... a tool the user 
will come to rely on" (p. 4). The second characteristic is the integration of evaluation into the 
design process. Evaluation should lead the process through the development of conceptual 
prototypes that allow for early assessment and redesign. 

The inclusion of a range of stakeholders early in evaluative activities is the third 
characteristic and allows for the collection of ideas, gains buy-in and commitment, and avoids 
unforeseen technical and administrative problems (Rathbun and Goodrum, 1994, p. 4). The 
range of stakeholders should include users, designers, local area network administrators, and 
others who impacted on the users' access to the program. The fourth characteristic is a 
suggestion that multiple collection techniques will produce a more complete evaluation than 
will a single source. This suggestion follows Lincoln and Guba's (1985) proposal that 
triangulation is a method of improving the probabilities "that findings and interpretations 
will be found credible" (p. 305). Denzin (1978) identified four basic types of triangulation: (1) 
data triangulation or the use of a variety of data sources; (2) investigator triangulation using 
several different researchers or evaluators to look at the same phenomena; (3) theory 
triangulation where multiple perspectives are used to interpret a single set of data; and (4) 
methodological triangulation where multiple methods are employed to study a single 
problem. The development of multiple techniques is both a sound and viable approach to the 
assessment of multimedia program development and implementation. 



Measuring the effectiveness of implementation focused on seeking the answers to four 
questions. The first question dealt with the usability/ accessibility of the multimedia program 
by students. Usability suggests that the end user finds the program adaptable to their level of 
competence and experience. Another way to state usability is to imply that users establish a 
comfort level with the program's operation. The second part of the first question is 
accessibility and refers to an availability of the program and appropriate hardware to users at 
times and locations that are convenient to the student's desired schedule. If the program is 
not readily accessible to students at peak user times or if the hardware cannot support the 
demands of delivery of the program at peak periods then the notion of accessibility is rejected. 
A Quicktime video delivered over a local area network needs to provide the user with 
unfettered delivery of the video. If the video is fragmented or choppy then accessibility could 
be considered compromised. 



The second question is receptivity by users. Another way to explaining receptivity suggests it 
is a process of valuing on the part of the user. Schon and Bennett (1996) suggest that good 
programs are "genuinely interactive and conversational" (p. 181). Rathbun and Goodrum 
(1994) stated "a tool viewed as relevant, critical, and of wide applicability is a tool users will 
come to rely on" (p. 4). They suggest valuing creates a face validity of the program. Chickering 
and Ehrmann (1996), reporting on good practices in education view technology as a lever in 
achieving greater student participation and engagement in learning. Student receptivity, as 
defined in this paper, seems to increase according to students' perceptions of how "hands-on" 
a learning activity is. They go on to suggest that active learning falls into "three categories: 
tools and resources for learning by doing, time-delayed exchange, and real-time conversion" 
all currently supported by software. 



Implementation readiness, the third question, deals with the order and timing of the 
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presentation of information. Structured within the context of the course the relevancy of the 
multimedia program is, in part, contingent upon appropriate implementation. 
Implementation includes some levels of valuing on the part of users, but more directly 
relates to student readiness to received what the instructor perceives to be appropriately timed 
and ordered material. The instructor/ designer has meaningful control over the 
implementation readiness. It is not the technology that is important, but how the technology 
is used to supplement and/or enhance learning (Ehrmann, 1996). 

The fourth question asks if the program met the instructional goals established by the 
instructor. It would seem apparent that any multimedia project would be designed to meet or 
support instructional goals. Sometimes, however, in the rush to adopt a new instructional 
methodology instructors forget the purpose and role of the technology. 

MEASUREMENT 



Students in the senior level course using the LT were expected to have had at least one course 
introducing them to computers and to have moderate to advanced experience with 
computers. The class was organized around one lecture and one discussion each week. The 
lecture section was composed of 60 students and the discussion groups of 20 students each. 

The lecture section typically presented new material or reinforced material presented in the 
LT. The discussion groups were utilized to expand lectures through discussion, involve 
students in practical exercises and discuss implications of information presented in the LT. 
The LT was intended to provided a foundation for classroom activities. 

Implementation of the LT was made via two modes: (1) across a campus-wide local area 
network; and (2) via CD. It was determined prior to the start of the semester to test the two 
implementation modes to see if there was any difference in the usability by the two groups 
and to determine if one mode was more preferable. 

Issues of time, resources, convenience, timing issues and the like precluded a complete and 
thorough pretesting prior to the implementation of the LT. The first semester of 
implementation of the LT more appropriately represented field testing of the final stages of 
development. Both the design team and instructor were aware that several questions about 
implementation remained unanswered and were likely not to be answered in the absence of a 
full implementation of the program. 



Triangulation was used to answer the 4 questions posed about the effectiveness of 
implementation. Figure 1 illustrates the methodologies utilized in the assessment. Two of the 
methods involved observation of students using the LT. In one case multiple students were 
observed using the multimedia program in a laboratory session. In this instance students 
were observed working through assigned tasks and the observer noted the frequency with 
which certain tasks were repeated, how students went about using the LT, time it took for 
students to complete a unit, and interactions between students. The second observation 
involved two volunteers from the class working individually with an observer through one 
LT unit. The third data collection method involved the use of two surveys. One was 
administered 5 weeks into the semester and a second 2 weeks before the end of the semester. 

A fourth method of analysis involved periodic debriefing the instructors following classroom 
use of the LT. A fifth artifact was the use of a critical incident questionnaire (Brookfield, 1995) 
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which was administered periodically during the course. The critical incident questionnaire 
was not designed to secure information about specific aspects of the LT implementation, but 
provided additional insights into student perceptions of the usability and receptivity of 
students. 



Assessment Question 


Assessment Methodology 


Usability of the multimedia by 
students 


Observation of group in computer lab 
Observation of individual students 
Survey instrument 
Debriefing of faculty 


Appropriate delivery of the 
technology to students 


Observation of group in computer lab 
Observation of individual students 
Survey instrument 
Debriefing of faculty 


Student receptivity to the use of 
technology to support classroom 
instruction 


Observation of individual students 
Survey instrument 



Figure 1: Assessment questions and methodologies 
RESULTS AND DISCUSSION 



USABILITY AND ACCESSIBILITY 



Perceptions of usability/accessibility of the multimedia project varied considerably. Two 
factors contributed to student perceptions of usability. Student knowledge of how to use 
computers and their comfort levels with them was a key factor. The availability of computers 
at a time that was convenient to the student was also very important. 

The usability of the multimedia project was clearly dependent upon student comfort and 
knowledge of computers. Even though most student were within 2 semesters of graduation 
(70%) several indicated they were not comfortable with computers. One student who 
volunteered to test the LT indicated he had a low comfort level with computers. This was 
evidenced when he reported that his e-mail did not work, and that he checked it only 
infrequently. He found the multimedia program to be informative and helpful, but had 
difficulty figuring out how to access it on the local area network. By contrast, another student 
volunteer reported himself as an above average computer user. He found the LT to be easy to 
access and to be helpful. In a separate survey it was determined that almost 50% of the student 
in this senior level course had yet to complete the required sophomore level computer 
applications course. This unfamiliarity with computers may have led to only 20% of the 
students in the class suggesting the multimedia project was a good use of their time. Another 
36% had no opinion. 



Unfamiliarity with computers was only one of several issues that affected perceptions of 
usability. Some students reported the computer freezing up when they attempted to access 
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Quicktime movies on a local area network during peak periods. Others reported software 
failure (most notably Netscape 2.02) when trying to access those portions of the course on the 
WWW. 

Accessibility was an equally difficult issue. Students indicated a problem with the Macintosh 
only format of the LT. This was evidenced when 51% of the students reported spending 30 
minutes or less each week on the LT. During parts of the semester the computer clusters were 
full with waiting lines. One observation made was the unwillingness of some students to 
access the LT during non-peak periods. The students suggested they would prefer access to the 
computer based materials through the use of a CD, used at home, or through the WWW. 

RECEPTIVITY 

Several valuing issues were apparent from student responses to the questionnaires. Students 
agreed that those LT units that implemented video aspects to demonstrate management 
processes were an aid to their understanding of management tasks and functions. In the first 
survey 41% of the students felt the scenarios were authentic and depicted the situational 
nature of the manager's job. In the second survey the same question was asked, but asked in 
conjunction with the worksheets provided in the student workbook. In this case 58% of the 
students valued the use of the LT, perhaps suggesting that learning by doing and allowing for 
reflection were important aspects of the implementation process. In addition to the 
Quicktime based scenarios, written scenarios were present on the WWW and were organized 
in a script format. Fifty-four percent of the students agreed the web scenarios were an aid to 
understanding the issues discussed in class. Maybe most importantly was an overwhelming 
recognition by students of the relevance of the LT units content to their future work lives 
(71%). 

The more often students used the LT and other computer based materials for the course the 
more comfortable they were with the process. The frustration level among students was 
much higher earlier in the semester when the process was new to almost all of the students. 
Later in the semester, the frustration level was significantly reduced. Unfortunately, students 
who became highly frustrated with the process and exhibited low expectations of their own 
ability to work with computers had a tendency to become disengaged from this aspect of the 
course. 



One of the late decisions made prior to implementation was the development of a workbook 
that provided much of the written information found in the LT and included questions, in a 
WWW format, guiding students through the video and WWW portions of the project. 
Clearly the addition of written materials designed to supplement and reinforce the LT 
presentation aided student understanding. 

There was a dichotomy between the usability/ accessibility and the receptivity of the course 
among the students. In the former, students found the software relatively unusable and non 
accessible. Receptivity did not suffer from the same perceptions. It was clear the students 
valued those portions of the content that connected with their perception of what the real 
work world is like and how they might work in it. The process of valuing may, in part, have 
encouraged some students to overcome the usability/ accessibility issues. This area obviously 
deserves more discussion. 
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IMPLEMENTATION 



Schon and Bennett (1996) reported, "It was amazing how much difference there was between 
the intentions that the faculty had for their software and the experiences that people had 
using it" (p. 179). This was certainly the case with the LT. Some implementation issues were 
structured by the availability of technology on campus. Others, however, were dictated by the 
willingness of students to use technology and their perceptions of the appropriateness of the 
technology to their future. It was also clear, that in some instances, the student outcomes were 
frequently not consistent with the expectations of the instructor. 

The assessment carried over two semesters and in those instances where it was determined 
the content was not valued or it did not contribute to student learning, the instructors 
modified the sequence, timing and delivery, and in several cases this made a significant 
difference in student receptivity. For example, in a unit where students observed a day in the 
life of a manager they found it an aid in their understanding of how managers function. The 
video segment was designed primarily to foster understanding among students with limited 
knowledge of what managers do. It was suggested that students with a broader base of 
management experience found this segment less helpful than those with a limited experience 
base. Debriefing the instructors (a graduate assistant facilitated one of the discussion groups) 
following the presentation of this particular Quicktime video, it was determined that during 
the second semester of implementation to stop the video more frequently for discussion. In 
addition the student workbook was modified so students were asked to look at more concrete 
types of roles managers engage in and then to think about why they thought those roles 
appeared. A critical incident questionnaire administered during the second semester had 
responses suggesting a higher level of understanding among the students. 

In another unit focusing on management theory students were exposed to the same scene 
portrayed in 3 different management styles. The resulting discussion comparing the 3 
management styles was confusing for the students. They had difficulty determining how the 
management styles were different. During the debriefing the instructor suggested that looking 
at short segments of 3 different management approaches did not provide students with 
sufficient information to gain an understanding for a particular management approach. Each 
management style was originally developed with 5 scenes. The following semester the 
instructor showed all 5 scenes from one management style followed by workbook based 
questions and classroom discussion. Then the students viewed a second style and the same 
learning approach was repeated. Student understanding of individual management styles and 
the differences between styles improved considerably. 

INSTRUCTIONAL OBJECTIVES 



Measuring the effectiveness of accomplishing instructional objectives was clearly more 
difficult. At the outset of the project several goals were established. These were previously 
reported in the discussion of the LT. The first goal was to provide students with an interactive 
learning environment that would enhance rather than replace the existing classroom. It was 
found that in some cases this goal was met and in other cases the goal was not met. Students 
did not perceive the usability/ accessibility of the LT to be satisfactory, but did value the 
content of the video and web-based management exchanges. The goal was partially achieved. 
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but considerable additional work needs to be done in order to fully attain this goal. 

A second goal of the LT was to build the individual student's base of professional experience. 
The accomplishment of this goal was more difficult to assess. For students who had minimal 
experience with management it seemed that the LT had the potential for building an 
individual student’s management experience level. For students who had, what they 
perceived to be, management experience, the evidence was less compelling. More disturbing, 
however, may be those students who did not believe that the scenarios presented realistically 
portrayed management tasks or roles. 

The third goal focused on enriching student understanding of management as a dynamic 
rather than static process. The assessment process did not measure accomplishment of this 
goal. The instructors perceptions of achievement of this goal was mixed. It was felt that more 
attention needs to be given to this goal before any type of assessment could be made. A fourth 
goal of the LT project focused on enhancing basic and intermediate computer skills. Using the 
LT as a foundation, students were also exposed to electronic communications software, the 
WWW, word processing, spread sheet and graphics software. 

REFLECTIONS AND FUTURE ORIENTATIONS 



The evaluation paradigm suggested by Rathbun and Goodrum (1994) provided a starting 
point for our evaluation of an instructional multimedia program. The adoption of 
triangulation in its various forms as suggested by Denzin (1978) proved to be a viable and 
potentially effective tool for assessment in this situation. Assessment methods should be 
unobtrusive and most importantly should be used continuously in order to provide timely 
information which can be acted on quickly. Using a mix of qualitative and quantitative 
methods and a variety of researchers helped lessen the burden placed on students, who often 
"burn out" on course evaluations. The mix of methods used here resulted in students' greater 
engagement in course content, instructors' increased interest in and improvement of 
instructional use of materials, and a rich pool of ideas and experiences to use in the 
refinement and improvement of the product and its future use. 



Several implementation issues seemed to dominate the overall context of the assessment. 
The usability /accessibility of a program is a key factor in producing receptivity. It was evident 
that in the absence of good usability/ accessibility students would not struggle to find 
relevance. On the other end of the scale, however, those who found relevance tolerated, but 
did not like, the poor usability/ accessibility of the LT. 



Related to the usability/ accessibility is a suggestion for a more intense early linkage with 
stakeholders would have been beneficial. Software promises and hardware anticipations 
didn't materialize, thus reducing the intended delivery capabilities of the LT. A software 
package was promised that would be cross-platform. Two and one-half years later the 
promised software is still nowhere in site. Similarly, close cooperation with the hardware 
stakeholders suggests development of a multimedia program that meets the lowest common 
denominator. One that can run on an old 486 as well as the new speedier machines. 
University computer clusters available to students do not always have the most current 
hardware. The developers need to make sure the software is developed for the least capable 
computer it will potentially be delivered on. 
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The learning objectives of the course need to be realistic, based on teaching and design 
experience, subject to trial and error (learn by doing), and constantly be tested. Many authors 
have suggested that software development is typically a long term process. With the impact of 
changing technology, increased demands on the class and software, changes in the design 
team and the like, it is easy to lose sight of the instructional objectives. While it is believed in 
this process sight was not lost of the educational objectives, too little trial by error and 
continuous testing was done. There were several factors that mitigated against this and are 
likely to occur in most settings of this type. However, in retrospect, ways to do more testing 
should have been integrated into the design process. It in no way guarantees that the 
outcomes would have been any different, but it does suggest that the surprises might have 
been lessened. We suggest a good motto is to experiment, experiment, experiment until you 
find what works and don't apologize. 

The video and WWW segments of the LT were valued by the student users. To be valued, 
however, they must be connected with classroom discussions and student workbooks. The 
linkage between the workbooks and the video and WWW segments was not fully explored, 
but observations of class discussions suggested that for those students who took the time to 
complete the workbooks there was a stronger engagement in the classroom discussion and 
with the LT. 

Consistent with the above finding was the observation that the students preferred to learn by 
doing. Learning by doing is an effective active learning technique. When linked with small 
group discussions it enhances reciprocity and cooperation among students. The LT was 
designed as a support to a learn by doing classroom and independent process. 

What to do next has, in part, been driven by the assessment process. In those instances where 
students did not find relevance in the LT we need to determine why. In several instances a 
dissonance was present between the anticipated outcome of the LT unit and the students' 
perception of the same unit's value to them during the course and into the future. It may be 
that a more careful review of the existing units that did work and a determination of why 
they worked will provide important clues about how to improve those units that did not 
work. 

A second issue for future discussion relates to integration, which has to do with external 
requirements placed on students that are more challenging than the internal requirements of 
the LT. The goal is for the students to independently recognize the importance of the 
information contained in the LT as an aid in the achievement of some authentic task. While 
this work has been going on, there has been no organized effort to address whether it is 
integrated with the LT or is a stand alone requirement without linkage. 
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